LLM Integration

Llama 3 integration for AI pair programming

Overview

Duet uses Cloudflare’s AI service with the Llama 3 8B Instruct model to provide intelligent pair programming assistance. The LLM generates responses based on conversation history and can output commands for sandbox execution.

Model Configuration

The worker uses Cloudflare AI binding configured in wrangler.toml:

[ai]
binding = "AI"

Source: ~/workspace/source/cf-worker/wrangler.toml:12-13

The model is accessed via the @cf/meta/llama-3-8b-instruct identifier:

private async runAI(messages: AIMessage[]): Promise<string> {
  const result = await this.env.AI.run("@cf/meta/llama-3-8b-instruct", {
    messages,
  });
  return result.response?.trim() || "";
}

Source: ~/workspace/source/cf-worker/index.ts:122-127

Message Format

The AI expects messages in a specific format:

interface AIMessage {
  role: "system" | "user" | "assistant";
  content: string;
}

Source: ~/workspace/source/cf-worker/index.ts:84-87

System Prompt

Duet’s AI behavior is defined by a system prompt that instructs it to be concise and use special tags for command execution:

const aiMessages: AIMessage[] = [
  {
    role: "system",
    content:
      "You are Duet, a concise pair-programming assistant. " +
      "You can run commands in a sandbox using <run>command</run> tags. " +
      "When asked to perform an action, briefly explain what you will do and wrap the exact shell command(s) in <run> tags. " +
      "Do NOT include predicted output in your response - just provide the explanation and command.",
  },
  // ...
];

Source: ~/workspace/source/cf-worker/index.ts:154-162

This prompt establishes:

• Identity: Duet is a pair-programming assistant
• Tone: Concise and action-oriented
• Command syntax: Use <run>command</run> tags
• Output handling: Don’t predict output, let sandbox provide it

Conversation Context

The AI receives context from recent conversation history:

const aiMessages: AIMessage[] = [
  {
    role: "system",
    content: "You are Duet, a concise pair-programming assistant..."
  },
  ...this.state.messages.slice(-10).map<AIMessage>((m) => ({
    role: m.role === "agent" ? "assistant" : "user",
    content: m.text,
  })),
  { role: "user", content: userMsg.text },
];

Source: ~/workspace/source/cf-worker/index.ts:154-168

The agent:

1. Includes the system prompt
2. Adds the last 10 messages from conversation history
3. Appends the current user message
4. Converts message roles (“agent” → “assistant”, “user” → “user”)

Command Execution Flow

The AI can trigger sandbox commands using special tags:

1. AI Response with Commands

The AI wraps commands in <run> tags:

I'll check the directory contents.
<run>ls -la</run>

2. Command Extraction

The agent extracts commands using regex:

private async executeCommands(text: string, roomId: string): Promise<string> {
  const matches = Array.from(text.matchAll(/<run>([\s\S]*?)<\/run>/g));
  let result = text;

  for (const match of matches) {
    const cmd = match[1]?.trim();
    if (!cmd) {
      continue;
    }
    // ...
  }
}

Source: ~/workspace/source/cf-worker/index.ts:185-193

3. Sandbox Execution

Each extracted command is executed in the room’s sandbox:

try {
  const sandbox = getSandbox(this.env.Sandbox, `sandbox-${roomId}`);
  const { stderr, stdout } = await sandbox.exec(cmd);

  const summary =
    stdout.slice(0, 500) || stderr.slice(0, 500) || "[no output]";
  result += `\n\nOutput (${cmd}):\n${summary}`;
} catch (e) {
  const msg = e instanceof Error ? e.message : String(e);
  result += `\n\nError (${cmd}):\n${msg}`;
}

Source: ~/workspace/source/cf-worker/index.ts:195-205

4. Output Appending

Command outputs are appended to the AI’s response:

• First 500 characters of stdout or stderr
• Error messages if execution fails
• “[no output]” if command produces nothing

5. Tag Removal

The <run> tags are stripped from the final response:

return result.replace(/<run>[\s\S]*?<\/run>/g, "").trim() || "";

Source: ~/workspace/source/cf-worker/index.ts:207

Complete Message Flow

private async handleMessage(roomId: string, rawBody: unknown): Promise<Response> {
  // 1. Validate request
  const parseResult = MessageRequestSchema.safeParse(rawBody);
  
  // 2. Create user message
  const userMsg: DuetMessage = {
    role: "user",
    userId: data.userId?.trim(),
    text: data.text.trim(),
    ts: Date.now(),
  };

  // 3. Build AI context
  const aiMessages: AIMessage[] = [
    { role: "system", content: "..." },
    ...this.state.messages.slice(-10).map(...),
    { role: "user", content: userMsg.text },
  ];

  // 4. Get AI response
  const text = await this.runAI(aiMessages);
  
  // 5. Execute any commands in response
  const textWithOutputs = await this.executeCommands(text, roomId);

  // 6. Create agent message
  const agentMsg: DuetMessage = {
    role: "agent",
    text: textWithOutputs,
    ts: Date.now(),
  };

  // 7. Update state
  const nextMessages = [...this.state.messages, userMsg, agentMsg].slice(-50);
  this.setState({ messages: nextMessages });

  // 8. Return response
  return Response.json({ reply: agentMsg.text, messages: nextMessages });
}

Source: ~/workspace/source/cf-worker/index.ts:129-183

API Request/Response

Request Format

{
  "text": "list files in the current directory",
  "userId": "user-123"
}

Validated by:

const MessageRequestSchema = z.object({
  text: z.string().min(1, "Text cannot be empty"),
  userId: z.string().optional(),
});

Source: ~/workspace/source/cf-worker/index.ts:9-12

Response Format

{
  "reply": "I'll check the directory contents.\n\nOutput (ls -la):\ntotal 48\ndrwxr-xr-x  12 user  staff   384 Mar  1 10:00 .",
  "messages": [
    {
      "role": "user",
      "userId": "user-123",
      "text": "list files in the current directory",
      "ts": 1709294400000
    },
    {
      "role": "agent",
      "text": "I'll check the directory contents.\n\nOutput (ls -la):\n...",
      "ts": 1709294401000
    }
  ]
}

Source: ~/workspace/source/internal/ai/client.go:42-47

Next Steps

• Sandboxes - Learn how commands are executed safely
• Durable Objects - Understand conversation state management
• Cloudflare Workers - See the request routing layer